Validating a Rapid Verification Method for Cognitive Diagnosis

If instructional techniques and formats need to be dynamically tailored to levels of learner expertise (Kalyuga, 2005A), a question of considerable interest is how to measure the levels of expertise rapidly and in real time. A rapid diagnostic approach has been developed to evaluate what is the highest level of organised knowledge structures (if any) a person is capable of retrieving and applying to the briefly presented task? With the rapid verification method, students were presented with a series of possible (correct and incorrect) intermediate steps of a task solution and asked to rapidly verify the correctness of each step. The rapid verification method was used for optimizing levels of instructional guidance in an adaptive computerbased tutor for vector addition motion problems in kinematics (Kalyuga, 2005B). To validate the rapid verification method, two experiments were designed to investigate if rapid verification tests of levels of learner expertise in two different domains would correlate highly with a more traditional evaluation of expertise based on observations of participants’ problem solving steps using video-recordings and concurrent verbal reports. 33 university students from Sydney participated in the study (in Experiment 1, data for one student was lost due to a software problem).